{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instance Segmentation with Point Prompts and SAM 3\n",
    "\n",
    "[![image](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/opengeos/segment-geospatial/blob/main/docs/examples/sam3_point_prompts.ipynb)\n",
    "\n",
    "This notebook demonstrates how to use the Segment Anything Model 3 (SAM3) for interactive instance segmentation using point and box prompts.\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install \"segment-geospatial[samgeo3]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from samgeo import SamGeo3, download_file, show_image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Sample Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://raw.githubusercontent.com/facebookresearch/sam3/refs/heads/main/assets/images/truck.jpg\"\n",
    "image_path = download_file(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_image(image_path, axis=\"on\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/7c155676-0e08-4d6d-b5b9-341edc6ccb6c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize SAM3\n",
    "\n",
    "To use point and box prompts (SAM1-style interactive segmentation), initialize SAM3 with `enable_inst_interactivity=True`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam = SamGeo3(backend=\"meta\", enable_inst_interactivity=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.set_image(image_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Masks by Point Prompts\n",
    "\n",
    "Select an object by clicking a point on it. Points are input in (x, y) format with labels:\n",
    "- 1 = foreground point (include this region)\n",
    "- 0 = background point (exclude this region)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Single foreground point - input as Python list\n",
    "sam.generate_masks_by_points([[520, 375]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sam.masks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sam.scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Point Prompts\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_points([[520, 375]], [1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/b7b549db-3cff-4a08-a190-634d19978df7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show the Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_anns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/93424ade-313c-4c6c-82ef-264461df3434)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_masks()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Masks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.save_masks(\"truck_mask.png\", unique=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Points with Labels\n",
    "\n",
    "Use multiple points to refine selection. Use label=0 for background points to exclude regions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Two foreground points on the truck\n",
    "sam.generate_masks_by_points([[500, 375], [1125, 625]], point_labels=[1, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_points([[500, 375], [1125, 625]], [1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/c157b713-c6b1-4ed3-a1b5-560f476c579c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_anns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/70e9943c-3466-4c68-8be7-8cb2787a86e7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_masks()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Background Points\n",
    "\n",
    "Add a background point (label=0) to exclude a region from the mask.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One foreground point on window, one background point on car body\n",
    "sam.generate_masks_by_points(\n",
    "    [[500, 375], [1125, 625]], point_labels=[1, 0]  # foreground, background\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_points([[500, 375], [1125, 625]], [1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/11c69b3a-5e82-43dd-aef1-bec7af5243f1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_anns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/3562f636-a569-426f-a752-612fad08359e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Box Prompts\n",
    "\n",
    "Use a bounding box in XYXY format (xmin, ymin, xmax, ymax) to select an object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Box around the front wheel\n",
    "sam.generate_masks_by_boxes_inst([[425, 600, 700, 875]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_boxes([[425, 600, 700, 875]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/72ac8f82-e4ef-49b9-94ae-42773f78a468)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_anns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/1d6d0bc8-2765-4782-9a3c-256ff01cba76)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Box Prompts\n",
    "\n",
    "Process multiple boxes at once for efficient batch segmentation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boxes = [\n",
    "    [75, 275, 1725, 850],  # Whole truck\n",
    "    [425, 600, 700, 875],  # Front wheel\n",
    "    [1375, 550, 1650, 800],  # Rear wheel\n",
    "    [1240, 675, 1400, 750],  # License plate area\n",
    "]\n",
    "\n",
    "sam.generate_masks_by_boxes_inst(boxes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_boxes(boxes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/cf940307-c94f-4790-bccb-094d48d2d588)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.show_anns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/user-attachments/assets/6a518352-9fc5-435a-954a-c233d419573e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sam.save_masks(\"truck_boxes_mask.png\", unique=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Low-Level API: predict_inst\n",
    "\n",
    "For more control, you can use the lower-level `predict_inst()` method which returns masks, scores, and logits directly. Input points and boxes can be provided as Python lists.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using Python lists for input\n",
    "masks, scores, logits = sam.predict_inst(\n",
    "    point_coords=[[520, 375]],\n",
    "    point_labels=[1],\n",
    "    multimask_output=True,\n",
    ")\n",
    "\n",
    "print(f\"Generated {len(masks)} masks\")\n",
    "print(f\"Scores: {scores}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show all masks with point overlays\n",
    "sam.show_inst_masks(masks, scores, point_coords=[[520, 375]], point_labels=[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Box prompt with Python list\n",
    "masks, scores, logits = sam.predict_inst(\n",
    "    box=[425, 600, 700, 875],\n",
    "    multimask_output=False,\n",
    ")\n",
    "\n",
    "sam.show_inst_masks(masks, scores, box_coords=[425, 600, 700, 875])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook demonstrated SAM3's interactive instance segmentation capabilities:\n",
    "\n",
    "**High-level API (recommended):**\n",
    "- `generate_masks_by_points()` - Generate masks from point prompts\n",
    "- `generate_masks_by_boxes_inst()` - Generate masks from box prompts\n",
    "- `show_points()` / `show_boxes()` - Visualize prompts\n",
    "- `show_anns()` / `show_masks()` - Visualize results\n",
    "- `save_masks()` - Save masks to file\n",
    "\n",
    "**Low-level API:**\n",
    "- `predict_inst()` - Direct access to masks, scores, and logits\n",
    "- `show_inst_masks()` - Display masks with overlays\n",
    "\n",
    "**Input formats:**\n",
    "- Points and boxes can be provided as Python lists or numpy arrays\n",
    "- Point labels: 1 = foreground, 0 = background\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "geo",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
